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Abstract 

As  a  part  of  the  MEOCS  2000  Project  to  plan  for  a  second  generation  of  the  Military  Equal 
Opportunity  Climate  Survey,  a  literature  review  was  conducted  focusing  on  recent  developments 
in  survey  management  technology  and  on  the  emergence  of  new  equal  opportunity  issues.  Studies 
of  the  advantages  and  disadvantages  of  computer-administered  surveys  versus  paper-and-pencil 
surveys  were  reviewed,  and  the  feasibility  of  adapting  the  MEOCS  to  a  computer  administered 
format  was  discussed.  Recommendations  were  made  for  changes  in  the  structure  of  MEOCS  and 
in  the  coverage  of  equal  opportunity  issues. 
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Toward  a  Second  Generation  MEOCS: 

Recommendations  for  Administration  Format  and  Issue  Coverage 

Since  1990  the  Military  Equal  Opportunity  Climate  Survey  (MEOCS)  has  been 
administered  to  service  members  and  civilian  employees  of  the  Department  of  Defense  (Landis, 
Dansby,  &  Faley,  1993).  The  MEOCS  has  become  a  primary  instrument  used  by  the  Department 
of  Defense  to  assess  equal  opportunity  climate  in  military  units.  This  report  discusses  the 
structure  and  issue  coverage  of  the  current  MEOCS,  examines  contemporary  literature  on  survey 
technology  and  administration,  and  explores  possible  directions  for  the  further  development  of  the 
MEOCS. 

Structure  of  the  MEOCS.  The  MEOCS  is  a  124-item  paper-and-pencil  inventory 
composed  of  six  parts  (Landis,  Dansby,  &  Faley,  1993).  In  the  first  part  respondents  are  asked  to 
estimate  the  likelihood  that  50  equal  opportunity  behaviors  occurred  in  their  unit  over  the  past  30 
days.  Likelihood  estimates  are  given  on  a  five-point  scale  ranging  from  "very  high  chance  that  the 
action  occurred"  to  "almost  no  chance  that  the  action  occurred."  Principal  components  analysis 
of  this  section  by  Landis  (1990)  revealed  five  factors  addressed  by  these  questions:  sexual 
harassment  and  discrimination,  differential  command  behaviors,  positive  command  behaviors, 
racist/sexist  behaviors,  and  reverse  discrimination. 

Three  short  parts  of  the  MEOCS  deal  with  respondents'  attitudes  towards  their  unit  as  an 
organization.  These  sections  address  levels  of  organizational  commitment,  work  group 
effectiveness,  and  job  satisfaction,  using  five-point  Likert  scales.  The  fifth  part  of  the  MEOCS 
consists  of  27  questions  adapted  from  the  Racial  Awareness  and  Perceptions  Scale  (RAPS),  an 
earlier  instrument  developed  by  the  Army  Research  Institute  to  measure  racial  attitudes  (Hiett, 
McBride,  Fiman,  Thomas,  O'Mara,  &  Sevilla,  1978).  Respondents  answer  using  a  five-point 
Likert  scale  from  "totally  agree"  to  "totally  disagree."  A  final  part  of  the  instrument  contains 
items  concerning  the  respondents'  personal  experiences  with  discrimination  as  well  as  items 
addressing  characteristics  of  the  respondent  and  the  unit. 

The  reliability  and  validity  of  the  MEOCS  have  been  discussed  by  Landis  (1990)  and 
Landis,  Faley  and  Dansby  (1992).  Landis  (1990)  reported  Cronbach's  alphas  ranging  from  .93  to 
.50  for  the  five  factors  of  the  MEOCS  part  1,  with  "racist/sexist  behaviors"  and  "reverse 
discrimination"  the  weaker  of  the  five  factors  (alphas  <  .70).  Landis,  Faley,  and  Dansby  (1993) 
examined  reliability  for  all  five  parts  of  the  MEOCS  and  found  alphas  above  .75  for  all  factors 
except  "belief  in  integration"  (alpha  =  .60).  No  diflerence  in  reliability  was  found  with 
respondents  in  lower  enlisted  ranks  versus  higher  ranks.  Dansby  (1996)  notes  that  the  average 
alpha  for  the  MEOCS  factors  is  .85,  better  than  that  of  most  organizational  climate  surveys. 
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Validity  of  the  MEOCS  has  been  more  difficult  to  assess.  The  MEOCS  is  intended  to 
measure  equal  opportunity  climate  in  military  settings;  however,  as  Dansby  (1996)  notes,  there 
are  few  other  quantitative  measures  of  equal  opportunity  climate  that  could  be  used  to  examine 
the  criterion  validity  of  the  MEOCS.  In  addition,  an  extensive  debate  continues  in  the 
management  and  organizational  psychology  literature  over  the  nature  of  organizational  climate 
itself  (Denison,  1996).  Despite  these  and  other  difficulties,  some  tentative  evidence  exists  that  the 
MEOCS  is  a  valid  measure  of  equal  opportunity  climate.  Individuals  were  given  a  description  of 
a  unit  with  "good"  or  "poor"  equal  opportunity  climate,  and  were  asked  to  complete  the  MEOCS 
for  the  unit  (Dansby,  1996).  MEOCS  scores  for  the  "good"  and  "poor"  units  differed  in  the 
predicted  direction.  Dansby  and  Landis  (1991)  also  report  evidence  for  construct  validity  in  initial 
field  administrations  of  MEOCS.  Results  for  demographic  subgroups  were  all  in  the  predicted 
directions. 

Versions  of  the  MEOCS.  Since  its  inception,  the  MEOCS  has  evolved  through  several 
versions.  Changes  include  addition  and  deletion  of  specific  items,  reordering  of  the  sections,  and 
development  of  a  universal  all-services  version  to  replace  separate  versions  for  each  service. 
Currently  four  versions  are  in  use:  The  standard  MEOCS  (version  2.3),  a  version  for  all-male 
units  (version  2.3M),  an  abbreviated  version  (MEOCS-LITE),  and  a  version  for  use  in  units  with 
civilian  employees  (MEOCS-EEO).  The  MEOCS-LITE  eliminates  questions  measuring  global 
attitudes  towards  discrimination  and  focuses  on  discrimination  issues  of  concern  to  the  local  unit. 
The  MEOCS-EEO  contains  additional  factors  which  address  civilian  equal  opportunity  issues 
such  as  discrimination  on  the  basis  of  age,  religion,  or  disability. 

Other  specialized  versions  of  the  MEOCS  have  been  developed  to  address  specific 
respondent  groups  and  situations.  The  Senior  Leader  Equal  Opportunity  Climate  Survey 
(SLEOCS)  assesses  the  equal  opportunity  beliefs,  opinions,  and  perceptions  of  general  and  flag 
officers  and  senior  civilian  leaders  (McIntyre,  1995).  A  Small  Unit  Equal  Opportunity  Climate 
Survey  (SUEOCS)  is  being  developed  to  assess  equal  opportunity  attitudes  of  units  with  50  or 
fewer  service  members  (Albright  &  Mclnt5n^e,  1995).  Finally,  a  civilian  version  for  use  in  college 
and  university  settings  also  has  been  developed  (University  Equal  Opportunity  Climate  Survey,  or 
UEOCS). 

Administration  process  of  the  MEOCS.  The  procedure  for  administering  the  current 
MEOCS  is  based  on  a  six-step  program  that  includes  initial  contact  with  a  commander, 
clarification  of  expectations,  collection  of  data,  analysis,  feedback  of  results  to  the  commander, 
and  follow-up  (Landis,  Dansby  &  Faley,  1993).  Once  a  request  is  made  to  DEOMI  for  an 
evaluation,  DEOMI  staff  determine  which  version  is  appropriate  for  the  unit.  A  camera-ready 
copy  is  sent  to  the  unit,  with  Scantron  answer  sheets  and  instructions.  Data  collection  is 
conducted  by  a  unit  project  officer  who  administers  the  survey  and  returns  the  answer  sheets  to 
DEOMI.  Data  are  analyzed  using  a  commercial  statistics  program  and  feedback  is  provided 
through  an  automated  report  generation  program  developed  at  DEOMI.  After  the  report  is  sent  to 
the  unit,  optional  consultation  and  training  are  provided  by  DEOMI  at  the  request  of  the 
commander. 


2 


Evaluation  of  MEOCS.  Although  a  systematic  evaluation  of  perceptions  of  MEOCS 
using  a  random  sample  of  respondents  and  administrators  has  not  been  conducted,  there  have 
been  content  evaluations  of  written  comments  on  the  MEOCS.  Grosch  (1994)  analyzed  658 
open-ended  written  comments  about  MEOCS  made  by  respondents  from  1990  to  July  1994. 
Comments  were  coded  into  perceptions  of  style  (the  "tone"  of  the  questions),  perceptions  of 
content  (the  substance  of  the  survey  and  the  topics  covered),  and  miscellaneous  (mostly  external 
factors  that  influenced  the  way  in  which  the  respondent  completed  the  MEOCS).  Grosch  found 
that  25%  of  the  comments  reflected  the  opinion  that  the  MEOCS  is  biased  toward  finding  a 
particular  outcome,  13.7%  felt  the  survey  was  offensive,  and  14.6%  felt  that  the  wording  of  the 
survey  items  was  not  appropriate.  However,  the  sample  for  this  study  represents  only  .3%  of  all 
MEOCS  respondents  in  that  time  period.  Given  the  extremely  low  percentage  of  respondents 
who  made  comments,  results  of  this  analysis  should  be  interpreted  with  caution. 

Hochhaus  (1995)  conducted  a  content  analysis  of  open-ended  comments  made  by  324 
respondents  on  the  Senior  Leader  Equal  Opportunity  Survey.  Hochhaus  developed  a 
categorization  scheme  for  written  comments  made  in  response  to  six  questions  concerning 
respondents'  beliefs  about  the  most  significant  EO  issues  facing  their  units,  perceptions  of  the 
strengths  and  weaknesses  of  their  service's  EO  programs,  and  opinions  about  the  most  important 
elements  of  an  effective  EO  program.  Factor  analysis  of  responses  indicated  that  respondents 
were  concerned  about  EO  backlash  issues  such  as  reverse  discrimination.  In  addition,  comments 
were  directed  towards  a  diverse  set  of  issues  including  support  of  EO  training,  belief  in  the 
importance  of  leadership  inEO  effectiveness,  and  sexual  harassment.  Although  Hochhaus'  study 
does  not  address  perceptions  of  the  MEOCS  as  an  instrument,  it  does  shed  some  light  on  senior 
leaders'  concerns  about  EO  issues  that  form  the  content  of  MEOCS. 

In  the  six  years  since  its  introduction,  the  MEOCS  has  expanded  into  a  family  of  EO 
surveys  that  assess  equal  opportunity  climate  in  a  wide  variety  of  settings  and  across  diverse 
respondent  groups.  The  surveys  have  been  well  accepted  by  commanders  and  now  are  standard 
tools  used  to  assess  equal  opportunity  climate  in  military  units.  Psychometric  studies  of  the 
MEOCS  surveys  show  them  to  have  a  stable  factor  structure  with  a  high  level  of  reliability. 
However,  despite  the  evident  success  of  the  MEOCS,  problems  remain.  As  Landis,  Dansby,  and 
Faley  (1993)  note,  mean  responses  on  the  MEOCS  tend  to  shift  towards  the  positive  end  of  the 
scale,  possibly  indicating  a  social  desirability  response  bias.  In  addition,  current  events  and 
changes  in  the  national  demographic  distribution  bring  to  attention  new  equal  opportunity  issues 
that  need  to  be  addressed.  Finally,  new  information  technologies  make  possible  administration  of 
the  MEOCS  in  different,  and  possibly  improved,  formats. 

MEOCS  2000 


Because  of  the  issues  noted  above,  a  long-range  planning  strategy  for  development  of  the 
MEOCS  recently  was  proposed  (McIntyre,  1996).  This  plan,  known  as  the  MEOCS-2000 
project,  proposes  to  evaluate  the  current  MEOCS  and  make  recommendations  for  implementation 
of  equal  opportunity  climate  assessment  in  the  military  after  the  year  2000.  As  described  by 
McIntyre  (1996)  the  proposal  has  four  parts: 
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j  MEOCS-2000  Proposal 

•  A  review  and  summary  of  the  psychometric  properties  of  the  MEOCS  to  assess  the  technical 
strengths  and  weaknesses  of  the  current  instrument 

•  A  survey  of  current  administrators  of  the  MEOCS  and  field  commanders  who  have  used  the 
MEOCS,  to  assess  the  perceived  quality  of  the  current  instrument  and  administration  process 

•  A  review  of  published  literature  on  equal  opportunity  climate  assessment,  focusing  on  recent 
technological  advances  in  survey  management  and  on  emerging  equal  opportunity  issues 

•  Use  of  the  World  Wide  Web  to  collect  information  on  survey  technology  and  contemporary 
EO  issues,  and  to  identify  EO  experts  for  interviews 


The  goal  of  the  MEOCS-2000  Project  is  not  to  replace  the  current  instrument,  but  rather  to 
identify  options  for  future  development  of  the  MEOCS.  The  current  report  focuses  on  parts  three 
and  four  of  the  MEOCS-2000  Project.  Questions  addressed  by  this  report  are: 


•  What  current  survey  management  technologies  exist  that  could  help  streamline  and  improve 
the  process  for  administering  the  MEOCS? 

•  How  might  the  structure  of  the  MEOCS  be  modified  to  address  an  increasingly  diverse  set  of 
EO  issues  and  demands? 


The  remainder  of  this  report  will  address  each  of  these  questions  in  turn,  and  will  conclude 
with  a  set  of  recommendations. 

Recent  Advances  in  Survey  Technology 

In  the  several  years  since  the  MEOCS  was  first  administered,  the  pace  of  development  in 
survey  technology  has  accelerated.  Although  paper-and-pencil  surveys  are  still  the  mainstay  of 
survey  research,  many  more  options  for  survey  administration  have  become  available  to 
researchers.  Use  of  rapidly  evolving  communications  technology  media  now  enable  researchers 
to  administer  surveys  by  fax,  in  CD-ROM  format  on  individual  personal  computers  (Dunnington, 
1993),  simultaneously  to  multiple  users  on  computer  network  servers  (Rosenfeld,  Doherty, 

Vicino,  Kantor,  &  Greaves,  1989),  by  electronic  mail  (Perlman,  1985;  Sproull,  1986),  or  on-line 
via  the  World  Wide  Web  (Read,  1991).  In  addition,  computer-assisted  personal  interviewing 
(CAPI)  and  computer-assisted  telephone  interviewing  (CATI)  have  expanded  the  use  of  surveys 
in  marketing  and  political  polling,  as  well  as  other  applications  (Baker,  1992;  de  Leeuw,  Hox,  & 
Snijkers,  1995). 

Although  computers  have  been  used  in  administration  of  psychological  instruments  for 
more  than  twenty  years  (Rosenfeld,  et  al.,  1989),  use  of  computers  for  large  scale  surveying  in 
organizations  is  a  more  recent  trend.  Researchers  have  been  attracted  to  the  use  of  automated 
data  collection  and  analysis  because  of  purported  cost  savings,  as  well  as  greater  efficiency  in  data 
collection,  analysis,  and  report  generation.  Large-scale  computerized  testing  is  now  routinely 
done  by  Educational  Testing  Service  in  administration  of  the  GRE  (Mead  &  Drasgow,  1993),  as 
well  as  by  many  corporations,  including  IBM  (Read,  1991),  AT&T,  John  Deere,  and  3M. 

Military  users  of  automated  surveying  include  the  Air  Force  Reserve,  Army  Research  Institute, 
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Naval  Pacific  Fleet,  and  the  Coast  Guard,  as  well  as  several  Air  Force,  Army,  and  Marine  bases 
(Raosofl,  1996;  Training  Technologies,  1996) 


Survey  management  software.  In  part,  the  burgeoning  popularity  of  automated  survey 
management  has  been  made  possible  by  the  availability  of  a  number  of  relatively  inexpensive 
survey  management  software  systems.  Dunnington  (1993)  reviewed  four  organizational  survey 
systems  available  at  that  time:  Survey  Software  Systems  (Insync  Corporation);  the  Organizational 
Universe  Survey  System  (Organization  Universe  Systems);  On-Line  Opinion  Survey  (IBM);  and 
Ci2  and  Ci3  (Sawtooth  Software).  A  recent  search  of  survey  and  polling  software  companies  on 
the  World  Wide  Web  uncovered  a  dozen  survey  management  packages  offered  by  various 
companies  (see  Appendix).  Typically  these  systems  offer  computer-assisted  survey  design, 
automated  data  collection,  and  automated  analysis  and  report  generation,  either  through  a  self- 
contained  analysis  package  or  by  export  of  the  survey  data  to  a  commercially  available  statistics 
program. 

Computer-administered  versus  paper-and-pencil  surveys 

Although  there  has  been  a  rapid  expansion  in  the  use  of  automated  survey  management 
packages,  research  on  the  effects  of  computer  administration  of  surveys  has  not  kept  pace  with 
developments  in  technology.  However,  an  expanding  body  of  work  is  beginning  to  address  the 
comparability  of  paper-and-pencil  versus  computer-administered  surveys  on  a  number  of 
dimensions,  including  the  equivalence  of  data  generated  by  the  two  methods,  differences  in 
respondents'  perceptions  and  behaviors,  levels  of  social  desirability  response  bias  with  the  two 
methods,  and  time  and  cost  differences. 

Data  comparability.  Can  data  from  a  computer-administered  test  be  interpreted  in  the 
same  way  as  data  from  a  paper-and-pencil  test?  Mead  and  Drasgow  (1993)  conducted  a  meta¬ 
analysis  of  159  cross-mode  correlations  from  29  studies  using  power  and  speeded  tests  of 
cognitive  abilities.  Speeded  tests  assess  an  individual's  ability  to  respond  to  a  homogeneous  set  of 
easy  items  in  a  limited  time,  whereas  power  tests  assess  an  individual's  ability  to  respond  to  items 
of  varying  difficulty  without  regard  to  time.  As  Mead  and  Drasgow  note,  two  kinds  of  problems 
may  emerge  in  changing  the  medium  of  administration  of  a  test:  (1)  scores  may  shift  either 
positively  or  negatively  when  the  medium  of  administration  is  changed,  and  (2)  the  psychological 
construct  being  measured  may  be  influenced  by  the  medium  of  administration.  The  second 
question  requires  relating  the  paper-and-pencil  and  computer-administered  tests  to  some  external 
criterion  to  compare  their  construct  validity.  Unfortunately  this  issue  is  not  addressed  by  most 
research  in  this  area. 

In  the  studies  of  cognitive  ability  testing  reviewed  by  Mead  and  Drasgow,  medium  of 
administration  influenced  the  equivalence  of  scores  on  speeded  tests,  but  not  on  power  tests. 

Mead  and  Drasgow  conclude  that  scores  from  computerized  and  paper-and-pencil  power  tests 
can  be  assumed  to  be  comparable;  however  a  change  in  the  administration  mode  of  speeded  tests 
should  be  undertaken  with  caution. 
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Other  studies  have  addressed  differences  in  administration  mode  in  noncognitive  tests. 
Kantor  (1991)  and  Rosenfeld,  et  al.  (1991),  for  instance,  found  equivalent  overall  levels  of  job 
satisfaction  with  computer  and  paper-and-pencil  administration  of  the  Job  Descriptive  Index. 
However,  Rosenfeld,  et  al.  (1991)  also  found  a  significant  administration  mode  by  self-monitoring 
interaction.  High  self-monitoring  respondents  reported  higher  levels  of  job  satisfaction  with  a 
paper-and-pencil  survey,  whereas  low  self  monitors  reported  higher  job  satisfaction  with  a 
computer-administered  survey.  Synodinos,  Papacostas,  and  Okimoto  (1994)  administered  touch¬ 
screen  and  paper-and-pencil  satisfaction  surveys  at  Honolulu  International  Airport.  No  significant 
difference  in  ratings  using  the  two  methods  was  found  among  randomly  selected  respondents. 
However,  self-selected  respondents  gave  more  negative  touch  screen  satisfaction  ratings  than  the 
randomly-selected  respondents.  Synodinos,  et  al.  interpret  this  finding  as  reflecting  motivational 
differences  between  the  self-selected  and  randomly-selected  respondents.  These  studies  suggest 
that  method  of  sample  selection  and  individual  differences  in  impression  management  may 
influence  the  comparability  of  data  from  computer-administered  and  paper-and-pencil  tests. 

Some  studies  have  investigated  the  influence  of  mode  of  administration  in  personality 
tests.  For  example,  Lankford,  Bell  &  Elias  (1994)  administered  a  battery  of  personality  measures 
including  the  Beck  Depression  Inventory  and  the  Purpose  in  Life  Test  to  undergraduates  using 
paper-and-pencil  and  computer-administered  format.  Subjects  who  had  been  pretested  as  high  in 
computer  anxiety  scored  higher  on  the  Beck  Depression  Inventory  and  lower  on  the  Purpose  in 
Life  Test.  Because  of  these  differences  between  high  and  low  computer  anxiety  subjects, 

Lankford  et  al.  conclude  that  standardized  norms  for  some  paper-and-pencil  personality  tests  may 
not  be  applicable  to  computerized  personality  tests.  King  and  Miles  (1995),  in  one  of  the  few 
studies  to  explore  the  factor  structure  of  paper-and-pencil  versus  computer  tests,  investigated  the 
similarity  of  underlying  factors,  factor  loadings,  and  mean  scores  on  the  Balanced  Inventory  of 
Desirable  Responding,  the  Mach  V  Scale,  the  Equity  Sensitivity  Instrument,  and  Rosenberg's  Self- 
Esteem  Scale.  No  evidence  for  an  effect  of  administration  mode  was  found.  Number  of  factors, 
factor  loadings,  and  mean  scores  did  not  differ  in  paper-and-pencil  and  computer  administration 
conditions. 

Some  design  considerations  in  survey  management  software  may  help  to  increase  the 
equivalence  of  paper-and-pencil  and  computer  surveys.  Flexible  computer  software  that  allows 
the  respondent  to  scan  the  survey  and  review  and  change  responses  increases  comparability  with 
paper-and-pencil  tests  (Spray,  Ackerman,  Reckase,  and  Carlson,  1989;  Whitener  &  Klein,  1995). 
However,  as  discussed  below,  allowing  respondents  to  review  and  change  responses  may  also 
increase  the  level  of  social  desirability  response  bias. 

In  summary,  most  studies  find  no  overall  difference  in  the  data  obtained  through  paper- 
and-pencil  surveys  and  computer  administered  surveys.  However,  there  are  a  few  caveats; 
speeded  tests  may  be  more  sensitive  to  changes  in  administration  mode,  and  equivalence  of  the 
data  may  be  influenced  by  respondent  characteristics  (such  as  self-monitoring),  by  whether  the 
respondents  were  self-selected  or  randomly  chosen,  and  by  the  features  of  the  computer  software 
program.  Few  studies  have  addressed  the  construct  equivalence  of  paper-and-pencil  versus 
computer  surveys. 
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Respondent  perceptions  and  behavior.  The  experience  of  taking  a  survey  by  computer  can  be 
radically  different  from  taking  a  paper-and-pencil  survey.  Some  computer  surveys  closely 
resemble  their  paper-and-pencil  counterparts,  but  in  other  computer  surveys  respondents  may  be 
forced  to  respond  to  one  item  before  the  next  one  is  presented,  may  be  presented  with  audio¬ 
visual  displays  including  sound  clips  and  video  images,  and  may  be  asked  to  respond  in  ways  that 
would  not  be  possible  with  a  paper-and-pencil  survey  (as  in  reaction  time  measurement).  Just  the 
experience  of  typing  a  response  rather  than  writing  it  may  influence  the  nature  of  the  response. 
Most  computer  survey  programs  allow  branching  of  questions,  so  that  each  subject  may  complete 
a  different  set  of  items.  All  of  these  differences  raise  questions  about  differences  in  respondents' 
perceptions  and  behavior  in  computer  surveys  versus  paper-and-pencil  surveys. 

Studies  of  respondent's  affective  reactions  to  computer  surveys  have  focused  on 
respondents'  levels  of  involvement  and  enjoyment  while  completing  computer  surveys.  For 
instance,  Rosenfeld,  et  al.  (1991)  noted  that  subjects  completing  the  Job  Descriptive  Index  on 
computer  reported  enjoying  the  experience  more  than  those  who  completed  the  paper-and-pencil 
version.  Locke  and  Gilbert  (1995)  report  similar  results.  When  undergraduate  students 
completed  a  short  form  MMPI  and  a  Drinking  Habits  Questionnaire  on  computer,  in  paper-and- 
pencil  format,  or  by  interview,  the  computer  format  was  perceived  most  positively.  Subjects 
reported  greater  comfort,  relaxation,  enjoyment,  anonymity,  and  confidence  in  the  computer 
condition,  and  less  fear,  boredom,  and  embarrassment. 

Other  studies  report  no  difference  in  respondents'  perceptions  of  computer  and  paper-and- 
pencil  surveys.  Kantor  (1991)  administered  the  Job  Descriptive  Index  to  176  Navy  civilian 
employees  either  on  personal  computer  or  in  paper-and-pencil  format.  As  reported  above,  levels 
of  job  satisfaction  did  not  differ  between  the  two  groups.  In  addition,  there  was  no  difference  in 
respondents'  enjoyment  of  the  two  formats  or  their  self-reports  of  the  truthfulness  of  their 
answers.  Finegan  and  Allen  (1994),  in  a  series  of  three  studies  using  a  battery  of  attitude  and 
personality  measures  found  no  difference  in  undergraduates'  levels  of  enjoyment  or  anxiety  in 
responses  to  computer  and  paper-and-pencil  surveys. 

Respondents'  levels  of  computer  literacy  and  computer  anxiety  play  a  part  in  affective 
reactions  to  computer  surveys.  Lankford,  Bell,  and  Elias  (1994)  pretested  subjects  for  state-trait 
anxiety,  mathematics  anxiety,  and  computer  anxiety.  Even  with  state,  trait,  and  math  anxiety 
partialed  out,  subjects  who  pretested  high  on  computer  anxiety  scored  higher  on  the  Beck 
Depression  Inventory  and  lower  on  the  Purpose  in  Life  Test  when  the  tests  were  administered  on 
computer  than  when  they  were  given  in  paper-and-pencil  format. 

The  impact  of  social  factors  on  respondents'  experiences  of  computer  and  paper  surveys 
should  not  be  minimized.  For  efficiency,  paper-and-pencil  surveys  ofl:en  are  administered  in  large 
group  settings.  In  contrast,  many  respondents  to  computer  surveys  complete  surveys  alone,  or  in 
the  company  of  a  small  number  of  other  respondents.  As  a  result,  most  studies  comparing  paper- 
and-pencil  surveys  with  computer-administered  surveys  confound  administration  mode  with  social 
environment  (Whitener  &  Klein,  1994).  An  extensive  body  of  research  on  social  influence 
suggests  that  social  environment  may  have  a  very  considerable  effect  on  respondents'  perceptions 
and  behavior  while  completing  surveys  (Cialdini,  1993;  Kiesler,  Siegel,  &  McGuire,  1984). 
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Unfortunately,  very  little  research  has  addressed  this  concern.  In  a  recent  study,  Whitener  and 
Klein  (1995)  administered  a  battery  of  personality  inventories  to  120  undergraduate  students  in  a 
3  (mode  of  administration)  x  2  (social  environment)  between-subjects  design.  Subjects  completed 
surveys  in  paper-and-pencil  format,  on  computer  with  no  scanning  allowed,  or  on  computer  with 
scanning  allowed.  In  addition,  half  the  subjects  completed  the  surveys  in  a  group  setting  and  half 
alone  in  an  office.  No  differences  in  need  for  achievement,  self-esteem,  of  locus  of  control  were 
found  across  the  three  modes  of  administration;  however,  subjects  reported  higher  levels  of  self¬ 
esteem  in  the  group  administration  condition  than  in  the  individual  administration  condition.  An 
interaction  between  mode  of  administration  and  social  environment  was  found  for  the  measure  of 
locus  of  control.  Although  somewhat  flawed  in  design,  this  study  is  a  first  step  towards 
examining  the  effects  of  social  environment  in  survey  administration. 

In  summary,  there  is  evidence  that  respondents  perceive  computer-administered  surveys  to 
be  at  least  as  enjoyable  and  involving  as  paper-and-pencil  surveys.  Given  the  relative  novelty  of 
computer  surveys  and  the  ability  of  computers  to  produce  colorful  and  d5mamic  stimuli,  this 
finding  is  not  surprising.  However,  respondents'  levels  of  computer  anxiety  and  the  social 
environment  in  which  the  survey  is  given  may  also  influence  subjects'  survey-taking  experience. 
Over  time,  as  the  computer  literacy  and  experience  of  respondents  increase,  these  results  may  be 
expected  to  change. 

Social  desirability  bias.  Are  survey  responses  given  by  computer  more  or  less  honest  than 
responses  to  paper-and-pencil  surveys?  The  problem  of  social  desirability  response  bias  haunts 
survey  research  of  all  types,  but  particularly  is  an  issue  in  surveys  such  as  the  MEOCS,  which 
address  sensitive  and  potentially  embarrassing  issues.  Respondents  may  be  less  than  candid  on 
surveys  in  a  number  of  ways,  including  giving  misleading  or  erroneous  responses,  refusing  to 
respond  to  questions,  or  underestimating  or  overestimating  responses  to  put  themselves  in  a  good 
light  (Kiesler  &  Sproull,  1986).  Such  biased  responding  may  arise  from  a  conscious  attempt  to 
hide  potentially  embarrassing  information,  or  may  result  from  respondents'  lack  of  insight  or 
subconscious  motives  to  protect  self-esteem.  Paulhus  (1984,  cited  in  Lautenshlager  &  Flaherty, 
1990)  has  labeled  these  two  forms  of  response  bias  “impression  management”  and  “self- 
deception.”  Paulhus  suggests  that  impression  management  is  more  likely  to  be  influenced  by 
factors  in  the  testing  situation  than  is  self-deception. 

Although  a  great  deal  of  research  has  been  directed  at  the  issue  of  social  desirability 
response  bias  in  computer-administered  versus  paper-and-pencil  surveys,  results  are  inconclusive. 
Computer-administered  surveys  initially  were  seen  as  one  way  of  reducing  social  desirabiUty 
response  bias.  Early  reports  suggested  that  computer  surveys  increased  candor  in  reporting  of 
sensitive  issues  such  as  alcohol  consumption,  sexual  behavior,  and  genito-urinary  S5anptoms 
(Locke  &  Gilbert,  1995;  Rosenfeld,  et  al,  1991).  Three  explanations  have  been  proffered  for  this 
effect;  (a)  On  paper-and-pencil  surveys  respondents  can  review  and  change  answers,  and  thus 
have  the  opportunity  to  alter  their  responses  in  a  socially  desirable  direction;  (b)  When  responses 
are  given  on  computer,  a  more  impersonal  and  anon5mious  social  situation  is  created  in  which  the 
person  is  less  concerned  about  appearances;  and  (c)  Respondents  on  computer  surveys  may 
believe  that  their  answers  are  being  monitored,  as  in  the  bogus  pipeline  effect  in  attitude  research 
(Rosenfeld,  et  al.,  1991). 
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Some  studies  have  found  that  respondents  give  more  extreme  responses  on  computer- 
administered  surveys  than  on  paper-and-pencil  surveys.  For  instance,  Feinstein  (1986,  cited  in 
Kantor,  1991)  found  that  respondents  completing  a  computer  survey  on  job  conditions  used 
harsher  language  in  criticizing  their  supervisors  than  those  who  completed  the  survey  in  paper- 
and-pencil  format.  Keisler  and  Sproull  (1986)  reported  that  respondents  who  completed  a 
computer  version  of  the  Need  for  Approval  scale  gave  fewer  socially  desirable  responses  than 
those  who  completed  the  paper-and-pencil  version.  These  findings  support  the  view  that  people 
are  more  self-absorbed  and  disinhibited  when  using  a  computer  than  when  completing  paper-and- 
pencil  surveys,  perhaps  leading  to  more  candid  responses.  The  social  anonymity  and 
depersonalized  nature  of  computer  communication  may  encourage  stronger  language  and  more 
assertive  responses  (Kiesler,  Siegel,  &  McGuire,  1984). 

A  more  recent  study  by  King  and  Miles  (1995)  examined  differences  in  factor  structure 
and  impression  management  in  computer-administered  and  paper-and-pencil  surveys.  Although 
no  differences  were  found  in  number  of  factors  or  in  factor  loadings  (as  noted  above),  impression 
management  was  higher  in  the  paper-and-pencil  surveys  than  in  the  computer  surveys.  This 
suggests  that  differences  in  impression  management  between  the  two  modes  of  administration 
cannot  be  attributed  to  measurement  inequivalence  of  the  instruments. 

In  contrast  to  the  research  just  described,  a  smaller  set  of  studies  have  found  increased 
social  desirability  bias  on  computer  surveys.  One  of  the  most  cited  of  these  is  a  study  by 
Lautenschlager  and  Flaherty  (1990)  in  which  241  undergraduate  psychology  students  completed 
Paulhus'  Balanced  Inventory  of  Desirable  Responding  (BIDR)  in  computer  or  paper-and-pencil 
versions.  Subjects  in  an  ''identified''  condition  were  asked  to  provide  their  name  and  other 
identifying  information,  and  were  told  that  their  survey  responses  might  enable  them  to  participate 
in  future  research.  Subjects  in  the  "anonymous"  condition  were  not  asked  for  identifying 
information  and  were  told  that  their  responses  would  be  anonymous.  In  contrast  to  previous 
studies,  greater  impression  management  was  found  in  the  computer  surveys  than  with  the  paper- 
and-pencil  surveys.  As  expected,  impression  management  in  the  anon5nnous  condition  was  lower 
than  in  the  identified  condition.  In  a  more  recent  study  by  Finegan  &  Allen  (1994)  using  three 
measures  of  socially  desirable  responding,  no  main  effect  for  administration  mode  on  socially 
desirable  responding  was  found.  However,  there  was  an  interaction  of  administration  mode  with 
computer  experience.  Respondents  with  little  computer  experience  who  completed  the  surveys 
on  computer  showed  higher  levels  of  socially  desirable  responding. 

A  third  set  of  studies  found  no  difference  in  social  desirability  response  bias  between 
computer  and  paper-and-pencil  surveys.  Millstein  (1987)  asked  female  adolescents  about  their 
sexual  behavior,  substance  use,  and  gynecologic  and  nongynecologic  symptoms  in  a  computer 
interview,  face-to-face-interview,  or  self-administered  questionnaire.  No  differences  in  responses 
were  found  across  the  three  administration  modes.  Booth-Kewley,  Edwards,  and  Rosenfeld 
(1992)  administered  two  scales  fi'om  the  BIDR  to  Navy  recruits  under  three  administration 
conditions:  computer-administration  with  backtracking  allowed,  computer  administration  with  no 
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backtracking,  and  paper-and-pencil.  Half  the  respondents  were  asked  to  provide  their  names  and 
social  security  numbers  and  half  completed  the  survey  anonymously.  No  main  effect  for 
administration  mode  was  found;  however,  in  support  of  Lautenschlager  and  Flaherty  (1990), 
impression  management  was  lower  in  the  anonymous  condition  than  in  the  identified  condition. 
Finally,  Locke  and  Gilbert  (1995)  found  no  difference  in  self-disclosure  among  undergraduates 
who  completed  a  short  form  MMPI  and  Drinking  Habits  Questionnaire  on  computer  or  by  paper- 
and-pencil  questionnaire.  However,  respondents  in  both  conditions  showed  higher  self-disclosure 
than  students  who  were  interviewed. 

Individual  differences  in  impression  management  might  be  expected  to  influence  socially 
desirable  responding.  A  popular  instrument  used  to  measure  impression  management  as  a 
personality  trait  is  the  self-monitoring  scale  (Snyder,  1974).  High  self-monitors  are  concerned 
about  social  norms  and  are  skilled  at  changing  their  behavior  to  fit  in  with  situational  expectations. 
Low  self-monitors  are  less  concerned  about  social  norms  and  act  more  according  to  internal 
values  and  beliefs.  Rosenfeld,  et  al.  (1991)  hypothesized  that  individuals  high  in  self-monitoring 
would  give  overly  positive  responses  on  a  paper-and-pencil  survey  of  job  satisfaction,  but  not  on 
the  computer  version.  Low  self-monitors  were  expected  to  be  unaffected  by  mode  of 
administration.  Results  supported  the  hypothesis  for  high  self-monitors,  but  not  for  low  self¬ 
monitors.  Low  self-monitoring  respondents  showed  higher  job  satisfaction  scores  on  the 
computer  survey  than  on  the  paper-and-pencil  survey. 

Although  the  research  on  social  desirability  response  bias  in  computer-administered  and 
paper-and-pencil  surveys  is  mixed,  it  is  clear  that  respondents'  ability  to  scan  and  to  modify 
answers,  individuals'  computer  experience  and  impression  management  tendency,  and  the  relative 
anonymity  of  the  survey  environment  all  contribute  to  the  pressure  towards  socially  desirable 
responding. 

A  number  of  strategies  have  been  used  by  researchers  to  limit  the  influence  of  social 
desirability  response  bias  in  surveys.  Techniques  include  use  of  anonymous,  self-administered 
questionnaires;  inclusion  of  forced-choice  questions  that  are  equivalent  in  social  desirability;  use 
of  the  randomized  response  technique  (described  below);  physiological  measurement  of  attitudes 
through  methods  such  as  pupil  dilation  and  galvanic  skin  response;  and  use  of  the  "bogus  pipeline" 
technique,  in  which  subjects  give  responses  while  connected  to  a  machine  which  supposedly  can 
detect  lies  (Rosenfeld,  et  al.,  1991).  Other  techniques  involve  hiding  the  sensitive  question  among 
innocuous  questions,  reducing  the  threatening  nature  of  the  sensitive  question  by  including  a 
statement  that  the  behavior  in  question  is  "normal,"  and  asking  the  respondent  how  others  might 
feel  about  the  issue  (Hosseini  &  Armacost,  1993). 

Two  of  the  most  recent  and  sophisticated  strategies  for  reducing  social  desirability 
response  bias  are  the  randomized  response  technique  (RRT)  and  use  of  neural  network  statistical 
techniques.  Hosseini  and  Armacost  (1993)  describe  in  some  detail  the  application  of  the 
randomized  response  technique  to  organizational  surveys.  In  one  of  the  more  popular  versions  of 
this  technique,  respondents  are  presented  with  questions  in  pairwise  fashion.  One  of  the  two 
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questions  is  innocuous  ("Is  your  birthday  in  February?")  while  the  other  addresses  the  sensitive 
issue  in  question.  Respondents  are  asked  to  answer  only  one  of  the  questions,  depending  on  the 
results  of  a  randomization  process.  For  instance,  a  respondent  may  be  asked  to  draw  a  marble 
from  ajar  for  each  pair  of  questions.  If  a  white  marble  is  drawn,  the  subject  answers  the 
innocuous  question,  and  if  a  red  marble  is  drawn  the  subject  answers  the  sensitive  question. 
Randomization  might  also  be  done  by  flipping  a  coin  or  using  a  spinner.  Once  the  data  are 
collected,  statistical  techniques  are  applied  to  estimate  the  proportion  of  respondents  who 
answered  in  a  particular  way  to  the  sensitive  question  (Hosseini  &  Armacost,  1993).  Several 
versions  of  this  technique  have  been  developed  using  different  formats  for  presentation  of 
questions. 

Although  cumbersome,  RRT  has  the  advantage  of  insuring  that  individual  answers  cannot 
be  attributed  to  a  particular  respondent,  thus  increasing  the  respondent's  confidence  in  his  or  her 
privacy  and  anonymity.  In  addition,  researchers  are  protected  from  legal  action  because  an 
individual  respondent's  answer  to  a  specific  question  cannot  be  known.  Research  using  RRT  has 
addressed  such  issues  as  abortion,  sexual  behavior,  and  drug  use.  In  organizational  settings  RRT 
has  been  used  in  surveys  of  employees'  compliance  with  financial  procedures,  and  in  studies  of 
shoplifting  and  illegal  use  of  company  resources  (Hosseini  &  Armacost,  1993).  The  advantages 
of  RRT  are  not  without  cost,  however.  RRT  studies  require  more  extensive  instructions  to  the 
respondent  about  the  procedure.  In  addition,  larger  samples  are  required,  and  the  aggregate 
nature  of  the  data  limits  the  kinds  of  statistical  techniques  that  can  be  used. 

A  recently  developed  statistical  technique  just  beginning  to  be  applied  to  surveys  is  the  use 
of  neural  networks.  Neural  networks  emerged  from  research  on  models  of  how  the  human 
nervous  system  learns,  but  have  evolved  into  a  general  process  for  deriving  patterns  and  meaning 
from  complex  and  imprecise  data  (Z  Solutions,  1996).  The  resulting  statistical  techniques  are 
similar  to  other  strategies  used  to  extract  relationships  from  large  data  sets,  such  as  multiple 
regression,  factor  analysis,  and  cluster  analysis  (Gatley,  1996).  Neural  network  techniques, 
however,  are  said  to  be  more  sensitive  to  interactions  and  interdependencies  in  the  data.  Because 
the  procedure  is  iterative,  evolving  through  a  series  of  layers  of  processing,  the  program  can 
"learn"  from  one  layer  to  another  as  it  converges  from  the  raw  data  to  a  solution.  For  instance, 
the  technique  might  be  used  to  predict  how  many  runs  will  be  scored  in  a  baseball  game,  using  the 
team's  offensive  statistics  over  a  series  of  games  (Z  Solutions,  1996).  Typical  applications  for 
neural  networks  have  been  in  engineering,  in  pattern  recognition  and  robotics.  However,  recently 
business  applications  have  emerged,  such  as  market  forecasting  and  detection  of  credit  card  fraud 
through  recognition  of  patterns  of  card  use  associated  with  stolen  cards  (Gatley,  1996).  A  Web 
search  on  neural  networks  revealed  several  companies  specializing  in  application  of  neural 
networks  to  survey  research.  At  least  one  such  company,  Delphi  Consultative  Surveys  and 
Research  (International)  Ltd.,  has  applied  these  techniques  to  the  study  of  racial  relations  in  the 
United  States.  It  is  claimed  that  this  technique  can  be  used  to  derive  descriptions  of  an 
organization's  equal  opportunity  climate,  while  asking  indirect  questions  that  would  not  arouse 
social  desirability  response  bias  (A.  S.  Gillman,  personal  communication,  August  16, 1996). 
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Time  and  cost  differences.  Although  computer-administered  surveys  offer  a  number  of 
advantages,  practical  considerations  of  time  and  cost  savings  need  to  be  considered.  Time  factors 
include  the  time  the  subject  uses  to  complete  the  survey,  data  entry  and  analysis,  and  report 
preparation,  as  well  as  mailing  delays.  Most  studies  comparing  computer-administered  surveys  to 
paper-and-pencil  surveys  find  that  respondents  complete  surveys  faster  on  computer  (Rosenfeld, 
Booth-Kewley,  &  Edwards,  1993).  Since  computer-administered  surveys  also  typically  involve 
automated  data  entry  and  analysis,  these  phases  of  the  survey  process  also  are  faster.  However, 
computer-administered  surveys  use  more  complex  and  vulnerable  technology  than  paper-and- 
pencil  surveys.  The  downtime  associated  with  computer  malfunctions  and  maintenance  has  not 
been  investigated  in  these  studies. 

Paper-and-pencil  surveys  are  usually  least  expensive  in  terms  of  materials  cost  and  mailing 
expenses.  The  cost  of  computer-administered  surveys  is  difficult  to  determine,  and  varies 
depending  on  the  hardware  and  software  available  to  the  researchers.  If  appropriate  computer 
equipment  is  in  place,  computer  surveys  may  not  be  more  expensive  than  paper-and-pencil,  and 
may  be  less  expensive  due  to  greater  efficiency  in  data  entry  and  analysis  (Rosenfeld,  et  al.,  1993). 
Often  cost  comparisons  do  not  take  into  account  the  expense  of  labor,  which  may  be  considerable 
in  the  preparation,  mailing,  and  data  entry  for  a  paper-and-pencil  survey.  Although  processing  of 
computer  surveys  likely  requires  more  skilled  labor,  costs  may  be  expected  to  be  less  since  the 
number  of  hours  required  to  process  the  survey  will  be  considerably  reduced.  Given  the  number 
of  variables  involved,  a  true  comparison  of  the  cost  and  time  involved  in  a  computer-administered 
versus  paper-and-pencil  survey  must  be  done  on  a  case  by  case  basis. 

Applicability  of  computer-administered  formats  to  MEOCS 

In  many  ways  the  MEOCS  surveys  are  not  t3^ical  of  surveys  administered  in 
organizational  settings.  They  are  part  of  an  ongoing  research  program  in  one  of  the  largest 
organizations  in  the  world— the  U.S.  Department  of  Defense.  They  are  given  only  at  the  request 
of  a  unit  commander  and  feedback  is  confidential  to  that  commander.  The  testing  situations  vary 
widely,  fi'om  field  units  to  command  offices,  and  the  respondents  vary  from  new  recruits  to  career 
senior  officers  and  civilian  employees.  Due  to  the  large  scale  of  the  MEOCS  program,  any 
decision  to  change  the  format  of  MEOCS  to  a  computer-administered  survey  would  carry  with  it 
potentially  great  benefits,  but  also  potentially  great  costs. 

Because  the  current  MEOCS  uses  scanned  answer  sheets,  data  entry  is  already  more 
efficient  than  unscanned  paper  surveys.  In  addition,  the  procedure  for  analysis  and  report 
preparation  of  the  MEOCS  has  been  automated.  Despite  these  advantages,  inefficiencies  exist  in 
the  administration  process.  The  master  form  of  the  MEOCS  is  mailed  to  a  unit,  photocopied  on 
site,  and  answer  sheets  are  returned  by  mail.  This  delivery  process  can  result  in  delays  of  a  week 
or  more  in  each  direction.  In  addition,  the  answer  sheets  must  be  hand  fed  through  a  scanner,  a 
process  that  is  both  labor  intensive  and  time  consuming.  Converting  to  a  computer-administered 
format  might  result  in  some  time  savings  in  these  areas.  If  a  disk-based  MEOCS  were  developed 
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to  be  administered  by  personal  computer  on  site,  data  could  be  in  a  format  ready  for  analysis  when 
the  disk  was  returned,  thus  eliminating  the  time  associated  with  scanning.  Data  from  the  disk 
could  be  returned  either  by  conventional  delivery  or  electronically.  Electronic  transfer  of  data 
would  also  reduce  delays  associated  with  mail  delivery. 

Computer-administered  surveys  are  capable  of  presenting  a  much  greater  variety  of 
questions  and  question  formats  than  paper-and-pencil  surveys.  In  addition  to  standard  text  and 
image  formats,  sounds,  narrations,  and  video  can  be  presented  on  computer  administered  surveys. 
Branching  of  questions  is  much  less  awkward  with  computer-administered  surveys  than  with 
paper-and-pencil  surveys.  These  features  would  give  greater  flexibility  to  the  design  of  future 
versions  of  the  MEOCS. 

Finally,  using  a  computer-administered  format,  it  may  be  possible  to  automate  the 
randomization  process  for  the  randomized  response  technique.  This  would  enable  development  of 
an  RRT  version  of  the  MEOCS  that  would  be  less  susceptible  to  social  desirability  response  bias. 
In  sum,  a  computer-administered  MEOCS  could  result  in  faster  processing  of  the  surveys,  more 
variety  in  question  design,  and  potential  reduction  in  response  bias. 

Several  notable  issues  and  problems  exist  in  converting  to  a  computer-administered 
MEOCS,  however.  First  among  these  is  the  availability  of  computers  in  the  units  where  the 
MEOCS  is  administered.  The  current  MEOCS  often  is  given  in  group  settings,  so  that  data  may 
be  collected  from  a  large  number  of  respondents  at  once.  The  speed  of  data  collection  with  a 
computer-administered  MEOCS  would  be  dependent  on  the  relationship  between  the  number  of 
computers  in  the  unit  to  the  number  of  personnel.  In  units  with  a  high  ratio  of  computers  to 
personnel  (such  as  command  offices)  data  collection  might  be  relatively  efficient;  but  in  units  with 
few  computers  and  a  large  number  of  personnel  (such  as  field  units)  there  would  be  a  substantial 
loss  in  the  efficiency  of  data  collection.  For  this  reason,  any  computer-administered  MEOCS 
would  not  be  suitable  for  all  units  in  which  the  MEOCS  is  currently  administered  and  therefore 
could  not  be  considered  a  replacement  for  the  paper-and-pencil  version.  As  computers  become 
more  ubiquitous  in  military  units,  this  may  become  less  of  an  issue. 

A  second  question  concerns  the  potential  computer  anxiety  of  MEOCS  respondents. 
Research  by  Lankford,  Bell,  and  Elias  (1994)  noted  above  suggests  that  when  respondents  are 
high  in  computer  anxiety,  data  from  a  computer-administered  survey  will  not  be  comparable  to  the 
paper-and-pencil  version.  It  is  likely  that  MEOCS  respondents  vary  widely  in  their  experience 
with  computers  and  their  level  of  comfort  with  computers.  Therefore  a  computer-administered 
MEOCS  would  be  most  applicable  in  units  in  which  the  personnel  are  high  in  computer  literacy 
and  there  are  a  sufficient  number  of  computers  to  allow  efficient  administration. 

Security  issues  also  would  be  of  concern  with  a  computer-administered  MEOCS. 

Although  it  is  possible  for  respondents  to  submit  multiple  or  fraudulent  copies  of  MEOCS  in  the 
paper-and-pencil  version,  this  possibility  is  reduced  by  administration  of  MEOCS  in  supervised 
groups.  If  MEOCS  were  administered  individually  on  computer,  it  might  be  easier  for 
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respondents  to  submit  multiple  copies.  In  addition,  a  breach  in  security  could  occur  when  data  is 
sent  electronically.  Use  of  a  password  system  for  respondents  and  encryption  of  data  that  is 
transmitted  electronically  may  lessen  some  of  these  concerns.  Some  of  the  current  commercially 
available  survey  management  systems  include  security  features  to  block  multiple  submissions  (see 
Appendix). 

It  is  clear  from  the  preceding  discussion  that  a  computer-administered  MEOCS  will  not 
replace  the  paper-and-pencil  version  in  the  short  term.  However,  given  the  potential  advantages 
of  a  computer-administered  format,  development  of  a  prototype  computer  version  of  MEOCS  to 
be  used  with  a  limited  set  of  units  may  be  desirable. 

Equal  Opportunity  Issue  Coverage  in  the  MEOCS 

The  MEOCS  originally  was  developed  to  measure  equal  opportunity  climate  for  minorities 
and  women  in  the  military.  In  1971,  the  Department  of  Defense  established  the  Defense  Race 
Relations  Institute  to  develop  a  training  program  in  race  relations.  Later,  in  1979,  the  name  of  the 
Institute  was  changed  to  the  current  Defense  Equal  Opportunity  Management  Institute  to  reflect 
an  expanded  mission,  including  training  on  equal  opportunity  issues  for  women  as  well  as  racial 
minorities  (Dansby  &  Landis,  1991).  This  change  in  focus  also  included  a  shift  in  emphasis  from 
concern  with  individual  racism  and  sexism  to  institutional  discrimination  on  the  basis  of  race  and 
gender.  However,  the  overall  mandate  of  the  Institute  remained  the  same:  To  improve  the 
readiness  and  effectiveness  of  military  units  by  minimizing  the  destructive  influences  of  prejudice 
and  discrimination.  With  this  objective,  the  MEOCS  was  developed  as  an  instrument  for 
diagnosing  the  racial  and  sexual  equal  opportunity  climates  of  units,  so  that  appropriate  training 
could  be  implemented. 

Rather  than  measuring  attitudes,  as  had  earlier  surveys  such  as  the  Racial  Attitudes  and 
Perceptions  Survey  (Hiett,  et  al.,  1978),  the  MEOCS  focuses  on  respondents'  perceptions  of 
discriminatory  behaviors.  This  approach  is  grounded  in  a  model  of  the  equal  opportunity  climate 
of  an  organization  as  a  combination  of  individuals'  perceptions  and  expectancies  (Dansby  and 
Landis,  1991).  In  Part  I  of  the  MEOCS,  respondents  are  asked  to  estimate  the  likelihood  that  .a 
series  of  discriminatory  behaviors  occurred  in  their  unit  within  the  past  30  days.  As  described 
above,  other  parts  of  the  MEOCS  address  respondents'  attitudes  as  well  as  perceptions  of 
organizational  commitment,  job  satisfaction,  and  work  group  effectiveness.  The  current  MEOCS, 
then,  measures  three  facets  of  equal  opportunity  climate:  Perceptions  of  discriminatory  behaviors, 
perceptions  of  the  organization,  and  racial  and  gender  related  attitudes.  Factor  analyses  of  the 
standard  MEOCS  show  it  to  have  twelve  factors  (DEOMI,  n.d.).  Factors  1  to  5  assess 
respondents'  perceptions  of  equal  opportunity  behaviors  in  the  unit,  factors  6  to  8  measure 
respondents'  perceptions  of  the  unit  as  an  organization,  factors  9  to  1 1  assess  respondents' 
attitudes  towards  general  racial  and  gender  issues,  and  factor  12  is  a  global  evaluation  of  the  unit's 
equal  opportunity  climate. 
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Factors  in  the  Standard  MEOCS  (version  2.3) 

Perceptions  of  Equal  Opportunity  Behavior 

1 .  Sexual  Harassment  and  Discrimination 

2.  Differential  Command  Behavior  toward  Minorities 

3 .  Positive  Equal  Opportunity  Behaviors 

4.  Racist/Sexist  Behaviors 

5.  "Reverse"  Discrimination  I  (at  the  local  unit  level) 

Perceptions  of  the  Unit  as  an  Organization 

6.  Commitment  (to  the  organization) 

7.  Perceived  Work  Group  Effectiveness 

8.  Job  Satisfaction 

Attitudes  Toward  Racial  and  Gender  Issues 

9.  Discrimination  toward  Minorities  and  Women 

10.  "Reverse"  Discrimination  II  (at  the  global  level) 

1 1 .  Attitudes  Toward  Racial/Gender  Separatism 

Global  Evaluation  of  the  Unit's  Equal  Opportunity  Climate 

12.  Overall  EO  Climate 


One  characteristic  to  note  in  this  factor  structure  is  the  intermingling  of  racial  and  gender 
discrimination  issues.  In  particular,  factors  4,  9,  and  1 1  concern  both  racial  and  sexual 
discrimination  issues.  The  content  of  some  individual  questions  in  the  MEOCS  also  reflects  a 
combined  concern  with  racial  and  gender  issues.  For  instance,  "Graffiti  written  on  the 
organization's  restroom  or  latrine  walls  'puts  down'  minorities  or  women"  (question  13).  Figure  1 
shows  the  distribution  of  questions  on  Part  I  and  Part  V  of  the  MEOCS  based  on  racial  and 
gender  content.  Although  questions  specifically  addressing  racial  issues  predominate,  a  substantial 
percentage  of  questions  in  both  parts  mix  racial  and  gender  issues.  This  "double  barreled" 
approach,  though  supported  by  factor  analysis  (Dansby  &  Landis,  1991),  complicates 
interpretation  of  data  from  these  questions  and  factors  (Zeisel,  1981).^ 

The  MEOCS  can  be  characterized  as  primarily  addressing  racial  issues  in  equal 
opportunity  climate,  with  gender  issues  a  secondary  concern.  The  intermixing  of  these  issues  in 
the  MEOCS  poses  several  problems.  First,  it  clouds  interpretation  of  scores  fi'om  factors  in  which 

*  Editor’s  Note:  This  has  been  recognized  by  DEOMI  researchers.  However,  the  concept  of  EO  climate,  which  is 
the  primary  concern  of  MEOCS,  is  expansive.  There  is  considerable  evidence  from  preliminary  development  of 
MEOCS  that  respondents  do  not  make  such  clear  distinctions  between  EO  climate  for  gender  and  EO  climate  for 
race.  In  early  development,  responses  to  over  100  items  representing  separate  (and  combined)  probes  for  gender 
and  race  issues  were  factor  analyzed.  Empirical  results  indicated  the  issues  were  often  highly  intercorrelated  as 
measures  of  EO  climate.  In  other  words,  respondents  tended  to  perceive  a  poor  racial  climate  and  a  poor  gender 
climate  as  occurring  together.  Though  it  would  have  been  possible  to  separate  scales  (e.g.,  Racist/Sexist 
Behaviors)  into  subscales  reflecting  race  and  gender  items,  it  was  thought  to  be  misleading  to  so.  Internal  studies 
have  shown  high  correlation  between  the  racial  and  gender  subscales  when  assessing  EO  climate. 
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racial  and  gender  issues  are  mixed.  If  a  unit  scores  unfavorably  on  one  of  these  factors,  does  it 
indicate  a  problem  in  racial  relations,  gender  relations,  or  both?  Second,  it  limits  the  flexibility  of 
the  MEOCS  in  addressing  specific  equal  opportunity  concerns  that  commanders  may  have.  For 
instance,  a  particular  unit  may  have  all  male,  but  racially  diverse  personnel.  As  currently 
constructed,  the  MEOCS  could  not  specifically  address  the  commander's  concern  with  racial 
issues  without  also  asking  about  gender  issues. 


Figure  1.  Distribution  of  questions  by  issue  in  MEOCS 
2.3,  Parts  I  and  V 


Despite  these  problems,  the  MEOCS  remains  one  of  the  most  successful  surveys  of  its 
kind.  Why  should  effort  be  directed  towards  a  major  redesign  of  the  MEOCS?  One  reason 
concerns  the  changing  demographic  population  from  which  the  military  draws  its  personnel.  As 
has  been  noted  in  the  public  media,  the  demographics  of  the  U.S.  population  will  change  radically 
in  the  next  fifty  years,  so  that  sometime  in  the  next  century  WTiites  will  no  longer  be  in  the 
majority.  The  increase  in  Hispanic  and  Asian,  as  well  as  African-American  and  Native  American 
segments  of  the  population  will  force  a  reconsideration  of  the  terms  "majority"  and  "minority." 
Rather  than  being  divided  primarily  along  black/white  lines,  we  are  becoming  a  multi-ethnic 
society.  In  order  to  remain  a  valid  measure  of  equal  opportunity  climate,  the  MEOCS  will  need 
to  change  to  reflect  these  trends. 

In  addition,  other  equal  opportunity  issues  emerge  from  time  to  time  that  may  need  to  be 
addressed.  A  recent  example  is  concern  over  possible  extremist  group  activity  in  the  military.  In 
December  1995  a  civilian  couple  in  Fayetteville,  North  Carolina,  was  murdered  by  three  soldiers 
based  at  Fort  Bragg  who  were  associated  with  a  local  "skinhead"  group.  Following  this  racially 
motivated  incident,  the  Secretary  of  the  Army  chartered  a  task  force  to  investigate  the  extent  of 
hate  group  activity  in  the  Army  (Department  of  the  Army,  1996).  The  task  force  initiated 
interviews  with  close  to  6,000  soldiers  at  28  bases,  and  administered  confidential  surveys  to 
17,000  other  soldiers. 
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On  the  basis  of  this  information  the  task  force  concluded  that  active  participation  in  hate 
group  activity  by  Army  personnel  was  minimal.  In  interviews,  fewer  than  one  percent  of  officers, 
noncoms,  and  civilian  employees  reported  that  a  soldier  or  civilian  employee  was  an  active 
participant  in  an  extremist  group,  and  also  less  than  one  percent  reported  having  any  contact  with 
extremist  groups.  In  contrast  to  the  interviews,  in  the  surveys  3.5%  reported  having  been 
approached  to  join  extremist  organizations  and  7. 1%  reported  knowing  another  soldier  whom 
they  believed  to  be  a  member  of  an  extremist  group.  The  latter  figure  was  discounted  by  the  task 
force  due  to  the  potential  for  multiple  reporting  of  single  incidents  on  the  surveys.  However,  data 
from  the  interviews  may  underrepresent  extremist  activity  since  the  interviews  were  carried  out  in 
groups,  thus  maximizing  pressures  for  response  bias. 

Although  active  participation  in  extremist  groups  by  Army  personnel  apparently  is 
minimal,  the  task  force  noted  that  extremist  groups  are  active  in  communities  surrounding  some 
Army  bases,  and  that  military  personnel  often  are  exposed  to  literature  from  extremist  groups.  As 
a  result,  the  task  force  made  several  recommendations  on  modifications  to  regulations  concerning 
participation  of  personnel  in  extremist  groups,  education  and  training  about  extremist  activities, 
and  screening  for  extremist  activity.  The  task  force  recommended  that  "automated  climate 
surveys  that  include  questions  on  extremism"  be  developed  (Department  of  the  Army,  1996,  p. 
29). 

Changing  the  structure  of  MEOCS 

As  currently  structured,  the  MEOCS  is  not  as  flexible  as  it  might  be.  The  intermingling  of 
race  and  gender  content  prevents  clear  interpretation  of  the  survey  results,  and  limits  the 
usefulness  of  MEOCS  for  units  with  specific  equal  opportunity  concerns.  In  addition,  the  current 
structure  does  not  lend  itself  readily  to  expansion  in  response  to  new  equal  opportunity  concerns 
such  as  extremist  activity.  For  these  reasons,  development  of  a  modular  MEOCS,  with  separate 
sections  for  racial  issues,  gender  issues,  and  other  concerns,  seems  desirable. 

Any  change  to  a  large  scale  survey  program  entails  certain  costs.  As  McIntyre  (1996) 
notes,  changing  MEOCS  would  involve  development  of  a  new  report  generation  system,  or  at 
least  modification  of  the  current  automated  system.  This  concern  might  be  lessened  if  a 
commercially  available  survey  management  program  could  be  adapted  for  MEOCS.  The  financial 
and  time  costs  in  moving  to  a  new  system  are  unknown,  and  would  need  to  be  explored. 

Changing  MEOCS  also  would  involve  retraining  administrators  and  educating  commanders  in  the 
field  about  the  new  system.  Any  new  system  would  necessarily  involve  a  break  in  the  continuity 
of  the  MEOCS  database.  Studies  would  need  to  be  done  to  determine  the  comparability  of  data 
from  the  old  and  new  versions. 

Instead  of  replacing  the  current  MEOCS,  a  more  modest  goal  would  be  to  develop  a 
modular,  perhaps  computer-administered,  MEOCS  as  a  supplement  to  the  current  survey. 
Inclusion  of  a  core  of  questions  from  the  old  MEOCS  would  help  to  preserve  the  continuity  of  the 
database.  Such  a  survey  might  be  made  available  to  units  that  meet  the  criteria  discussed  above. 
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Conclusions  and  Recommendations 

What  should  the  next  generation  of  MEOCS  look  like?  This  report  has  explored  the 
feasibility  of  two  kinds  of  change  to  the  MEOCS:  (a)  Change  in  the  administration  of  MEOCS  to 
a  computer-administered  mode,  and  (b)  change  in  the  structure  of  MEOCS  to  a  modular  format. 
In  addition,  new  techniques  for  reducing  social  desirability  response  bias  have  been  discussed. 
These  options  could  be  pursued  independently  or  together  in  a  prototype  second-generation 
MEOCS. 

Many  possible  scenarios  exist  for  the  development  of  MEOCS  into  the  next  century.  As  a 
first  step  toward  exploring  these  options,  a  systematic  survey  of  MEOCS  users  and  administrators 
should  be  conducted,  similar  to  that  specified  in  the  MEOCS-2000  proposal.  This  survey  could 
be  used  both  to  obtain  feedback  on  the  current  system  and  also  to  explore  the  feasibility  of 
modular  and  computer-administered  forms  of  the  MEOCS.  Respondents  should  be  asked  about 
the  quality,  informativeness,  and  ease  of  use  of  the  current  system,  and  about  the  relevance  of 
specific  EO  issues  to  their  work  setting  (McIntyre,  1996). 

It  is  strongly  recommended  that  a  modular  version  of  the  MEOCS  be  developed.  Such  a 
version  would  remedy  some  of  the  difficulties  in  interpretation  and  administration  discussed 
above,  and  would  increase  the  flexibility  of  the  MEOCS  in  adapting  to  future  EO  issues  that  may 
arise.  If  possible,  a  core  of  questions  from  the  old  MEOCS  should  be  retained  in  the  new  version 
to  facilitate  continuity  of  the  database. 

A  paper-and-pencil  version  of  the  MEOCS  should  be  retained  for  the  foreseeable  future. 
Because  of  the  diversity  of  settings  in  which  the  MEOCS  is  administered  and  the  varying 
computer  experiences  of  MEOCS  respondents,  no  computer-administered  version  of  the  MEOCS 
can  completely  replace  the  current  version.  However,  development  of  a  computer-administered 
version  of  the  MEOCS  should  be  explored,  both  as  a  prototype  for  future  MEOCS,  and  as  an 
option  for  units  that  are  able  to  utilize  it. 


Next  Steps:  Recommendations  for  Development  of  MEOCS 

•  Conduct  a  systematic  survey  of  MEOCS  users  and  administrators 

•  Develop  a  modular  MEOCS 

“  preserve  continuity  of  the  database  through  retention  of  a  core  of  questions. 

—  create  separate  modules  dealing  with  racial  issues, 
gender  issues,  and  extremist  activity 

•  Continue  support  and  use  of  a  paper-and-pencil  MEOCS 

•  Develop  a  protot5q)e  computer-administered  MEOCS 

•  Explore  use  of  the  randomized  response  technique  or  neural  networks  to  reduce  social 

desirability  response  bias _  ' 
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Finally,  new  techniques  to  reduce  social  desirability  response  bias  merit  exploration.  In  a 
survey  such  as  the  MEOCS,  which  asks  about  sensitive  and  possibly  embarrassing  issues,  the 
potential  for  bias  from  respondents'  evasion  and  misrepresentation  carmot  be  underestimated.  As 
has  been  noted,  responses  to  the  MEOCS  tend  to  be  shifted  to  the  positive  end  of  the  scale. 
Whether  this  indicates  the  presence  of  favorable  equal  opportunity  climates  or  is  a  symptom  of 
response  bias  is  an  open  question.  If  a  valid  assessment  of  equal  opportunity  climate  is  to  be 
obtained,  this  issue  must  be  addressed. 

The  evolution  of  survey  management  technology  and  changes  in  population  demographics 
will  inevitably  lead  to  changes  in  the  way  equal  opportunity  climate  is  measured  in  organizations. 
The  MEOCS  is  in  a  good  position  to  lead  the  innovation. 
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Appendix 


Survey  Management  Software 
CENSUS,  MASQ,  and  P-MASQ 

(Rosenfeld,  et  al.,  1989)  Three  versions  of  an  automated  survey  system  developed  at  the 
Naval  Personnel  Research  and  Development  Center  (NPRDC).  CENSUS  is  a  XENIX-based 
system  that  allows  simultaneous  entry  of  responses  from  16  users,  either  on-site  or  remotely 
through  modem.  Surveys  can  be  customized  to  individual  respondents  through  branching  of 
questions.  MASQ  is  an  MS-DOS  version  of  CENSUS  that  can  be  used  on  individual  stand¬ 
alone  microcomputers.  P-MASQ  is  used  on  portable,  laptop  computers.  Results  are  stored  in 
ASCn  files  for  export  to  data  analysis  programs. 

Decisive  Survey  for  Windows 

By  Decisive  Technology  Corporation  (http  ;//www.  decisive,  com/)  For  e-mail  surveys. 
Response  format  can  be  "choose  one,"  "choose  all  that  apply,"  ratings  or  text  responses. 
Allows  for  branching  of  questions.  Results  can  be  reported  by  question,  by  respondent,  or  by 
cross-tabs,  and  can  be  exported  to  SPSS  OR  SAS  for  further  analysis. 

System  requirements:  IBM  compatible  486/33;  8  MB  RAM;  Windows  3. 1  or  later;  15  MB 
hard  disk  space  plus  1  MB  per  100  respondents;  VGA  640x480  or  higher  color  display. 

Market  View 

By  Research  Systems  (http://www.resys.se/).  Available  in  a  Full  or  Lite  version,  as  well  as  an 
Academic  version.  Three  steps;  Questionnaire  design,  interviewing,  and  analysis.  Modules 
available  for  installation  on  a  local  network  or  for  distribution  as  a  diskette  survey. 

System  requirements:  Full  version  —  486  IBM  compatible;  Windows  3.x,  Windows  95,  or 
WindowsNT;  SMB  RAM;  7MB  disk  space.  Lite  version-- 386; ;  Windows  3.x,  Windows 
95,  or  Windows  NT;  4  MB  RAM;  3MB  disk  space. 

PinPoint 

By  Longman/Logotron,  UK  (http;//www.logo.com/pinpoint/).  Pinpoint  is  a  package  of  four 
programs:  a  questionnaire  designer,  a  data  collector,  a  database,  and  a  graph  and  chart 
generator.  Designed  primarily  for  e-mail  surveys,  but  data  may  also  be  entered  from  scanned 
forms  with  compatible  OMR  software.  Analysis  module  does  descriptive  statistics  and  chi- 
square;  can  export  data  to  statistics  programs. 

System  requirements:  386  IBM  compatible;  Windows  3. 1  or  higher  (Windows  95 
compatible);  4MB  RAM;  4MB  disk  space. 

Import/export  to:  CSV,  TSV,  Paradox,  dBase,  SPSS. 
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Sensus  Q&A  and  Ci3  for  Computer  Interviewing 

By  Sawtooth  Technologies  (http .7/www.  sawtooth,  com/products.html) 

Two  systems  for  design  and  administration  of  computer-assisted  interviewing  questionnaires. 
Can  create  disk-by-mail,  touchscreen,  audio  computer-aided  self-administered  interviewing,  as 
well  as  central  location  interview  questionnaires.  Has  multimedia  capabilities  to  include 
sounds,  narration,  and  video  in  surveys.  Allows  for  branching,  randomization,  and  response 
timing. 

The  Survey  Manager  for  Windows 

By  Insync  Corporation  (http://vell0cet.ins3mc.net/).  A  survey  management  system  that 
includes  survey  design,  administration,  analysis,  and  report  formatting.  Data  may  be  entered  . 
directly,  entered  online,  or  may  be  optically  scanned.  Reports  can  be  organized  by  question, 
by  group,  or  by  group/question.  Performs  descriptive  statistics  and  has  extensive  graphing 
capabilities. 

System  requirements:  IBM  compatible,  SMB  RAM,  Windows  3.1  or  later. 

Cost:  Single  PC  license:  $495;  Single  site  license:  $1,980;  Corporate  license:  $9,900. 

Survey  Said 

Distributed  by  Marketing  Masters 

(http://surveysaid.ostech.com:8080/marketing_masters/mmasters.htm).  Two  versions: 

Survey  Said  for  the  Web,  and  Survey  Said  for  Windows.  Survey  administration  software  for 
Internet  surveys,  electronic  surveys,  diskette  surveys,  kiosk  surveys,  and  paper  surveys. 

System  requirements: 

Cost:  $975.00  for  the  Web  version 

Survey  Select 

By  Saja  Software  (http://www.surveyselect.com).  Intended  for  business,  marketing,  and 
human  resources  applications.  Three  modules:  Design,  administration,  and  analysis.  Surveys 
can  be  built  from  templates  or  customized.  Generates  hard  copy  questionnaires  or  can  be 
administered  at  a  single  PC  or  over  a  PC  network.  An  Internet  version  is  expected  by  the  end 
of  1996. 

System  requirements:  486/DX  IBM  compatible  or  higher;  Windows  3.1  or  higher;  6MB 
RAM;  5MB  disk  space;  SVGA  with  512K  or  better. 

Maximum  number  of  records:  3000  per  survey 
Cost:  $595. 
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SurveyTracker 

By  Training  Technologies,  Inc.  (http://www.traintech.coni/).  Training  Technologies  provides 
Total  Quality  Management  consulting,  training,  and  software  services  to  business,  academic, 
and  governmental  clients.  The  SurveyTracker  software  package  includes  tools  for  planning, 
budgeting,  design,  administration,  analysis,  and  reporting  of  surveys.  Data  entry  can  be  from 
stand-alone  PC's,  via  a  network,  by  e-mail.  Responses  can  also  be  scanned  directly  from  the 
survey  form.  Computes  descriptive  statistics.  Chi-square,  and  correlation,  and  can  compare 
surveys  administered  at  different  times. 

System  requirements:  386SX  IBM  compatible  (486SX  recommended);  Windows  3.1  or 
higher  or  Windows  95;  8  MB  RAM;  25  Mhz  processing  speed  or  greater. 

Survey  Win  and  SurveyFirst 

By  Raosoft  (http://www.halcyon.coni/raosoft/).  SurveyWin  is  a  survey  management  package 
that  includes  a  questionnaire  builder,  data  entry  system,  database  management  tools,  and  a 
menu-driven  statistical  analysis  module.  SurveyFirst  is  a  stripped-down  version  of  Survey 
with  a  1,000  record  limit.  Data  is  stored  in  dBASE  III  format.  Analysis  module  includes 
descriptive  statistics,  t-tests,  chi-square,  correlation  and  regression.  Raosoft  Survey  is  being 
used  by  the  Army  Research  Institute  in  the  Occupational  Data,  Analysis,  Requirements,  and 
Structure  program  (ODARS), 

Cost:  $495.00  for  SurveyWin,  $195  for  SurveyFirst 

WWW  Survey  Assistant 

By  S  Ware  (http://or.psychology.dal.ca/~wcs/S_Ware.html).  S  Ware  is  a  company  formed  by 
a  Dalhousie  University  graduate  student  for  the  distribution  of  WWW  Survey  Assistant,  a 
shareware  Internet  survey  manager.  The  home  page  provides  a  comparison  table  of  WWW 
Survey  Assistant  with  other  Internet  survey  software  packages.  WWW  Survey  Assistant 
prepares  the  html  for  posting  a  survey  on  the  Internet,  and  creates  a  CGI  program  for 
managing  user-submitted  responses  to  the  survey.  The  program  allows  immediate  feedback  to 
survey  respondents.  Data  verification  and  manipulation  is  done  online  through  the  CGI 
program.  Survey  Assistant  can  filter  out  multiple  submissions,  and  has  security  protection 
features. 

System  requirements:  A  Web  server  and  access  to  CGI-bin;  an  HTML  Web  form  document, 
CGI  program,  and  Perl  programming  language. 

Cost:  $19.95  per  year  unlimited  use  site  license  for  individuals  and  not-for-profit  research 
organizations,  $99.95  per  year  for  for-profit  research  and  corporations. 
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